The present invention relates to a recording medium with video index information recorded therein, a video information management system which uses the video index information, a recording medium with audio index information recorded therein, an audio information managing method which uses the audio index information, a video retrieval method which uses the video index information, an audio retrieval method which uses the audio index information, and a video retrieval system. More specifically this invention relates to a recording medium with video index information available for retrieval of a movie and/or sounds, indexing of broadcast information, editing and preparation of a video program, provision of new information services to video information or for other purposes, a video information management method which uses video index information, an audio information management method which uses a recording medium with audio index information recorded therein and an audio information, a video retrieval method which uses the video index information with convenience in video retrieval further improved by video index information having a data structure with a logic structure of video information or audio information expressible therein, an audio retrieval method which uses audio index information, and a video retrieval system.
In recent years, in association with progress in the fields of computer-related technology or popularization of electronic networks such as the Internet, Intranet, Extranet or the like, all types of information used in daily life, business, and services are converted to electronic information. This information is then delivered to end users or accumulated in a computer-based database.
As described above, electronic information is used in various forms (including processing, retrieval, and delivery of information) irrespective of whether the information is digital one or analog one, and there have been proposed or actually put into use various types of technology for treating the electronic information itself with a format or a data structure enabling treatment thereof in an easier way.
For instance, text information, which is one type of electronic information, has a structure tag expressing a logic structure such as SGML, HTML, and XML to express the logic structure defined therein, and contents of information and a logic structure thereof can be expressed when the information is delivered. Therefore, when text information is to be used, processing for preparing and processing a summary enabling efficient retrieval of the text information can be executed at a higher speed.
In case of video information, which is another type of electronic information, there has been provided the video filing technology enabling efficient accumulation or retrieval of static images or movies.
Further as a conventional type of technology for retrieval of video-images, there have been disclosed the xe2x80x9cMedium For Retrieving Animation Image Information And Device Thereforxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 6-83877, xe2x80x9cMethod And Device For Video Image Retrievalxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 8-163479, xe2x80x9cAnimation Retrieving Systemxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 8-221432, xe2x80x9cVideo/Audio Raw Material Managing/Retrieval Systemxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 9-18432, xe2x80x9cMethod And Device For Retrieving Video Imagexe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 9-50436, and xe2x80x9cVideo Retrieval Methodxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 10-40260.
However, method of expressing a logic structure inside the video information for video information including sounds and animation is unknown. Therefore, there are problems such that the use of video information is not easy as compared to other types of information, or that there are several restrictions in reproducing the video information. In other words, there is the problem that there has not been provided the user interface technique enabling improvement in adaptability of video information to operation, application, and treatment.
More specifically, there have been such defects in the conventional technology as (1) retrieval can not be made with respect to the internal information in the video information, (2) jumping to a required section in order to identify a required scene can not be executed unless the entire video information is checked once and marks are put at required sections, and (3) a dedicated device or technology is required for preparing a digest of video information.
With the conventional technology for retrieving video-images as described above it is possible to retrieve video information including a desired scene from a plurality of video information (such as, for instance, a plurality of movie contents) or to identify a position at which a desired scene is included in one video information. However, the techniques can not always be applied to all types of video information because of the defects as described below.
With the xe2x80x9cMedium For Retrieving Animation Image Information And Device Thereforxe2x80x9d in Japanese Patent Laid-Open Publication No. HEI 6-83877, the operator is required to find out a desired scene, by looking at a movie index previously prepared by copying a sequence of moving pictures. In this case, there is the problem that a video-image obtained as a result of retrieval depends on subjectivity of the operator who searches visually by checking the moving picture index, and also that a large volume of moving pictures can not be treated.
With the xe2x80x9cMethod And Device For Video Image Retrievalxe2x80x9d disclosed in Japanese Patent Laid-Open Publication No. HEI 8-163479, it is required to successively display a series of images each showing a scene in video-information so that an operator can find out a desired scene from the displayed images. In this case, there is the problem that a video-image obtained as a result of retrieval depends on subjectivity of an operator searches visually by checking the moving picture index, and also that a large volume of moving pictures can not be treated.
With the xe2x80x9cAnimation Retrieving Systemxe2x80x9d disclosed in Japanese Patent Laid-Open Publication No. HEI 8-221432, a desired video is retrieved by using a menu in which videos as objects for retrieval are classified, so that a work for classifying the videos as objects for retrieval is required. Therefore, there is a problem that the precision in retrieval varies depending on how the videos are classified for preparing a menu.
The xe2x80x9cVideo/Audio Raw Material Managing/Retrieval Systemxe2x80x9d according to Japanese Patent Laid-Open Publication No. HEI 9-18432 is for a system used in a business organization or a public facility treating a large volume of video information such as a broadcasting station. Therefore, there is the problem that the information which can be treated with this system is limited to video information such as that for a news program for which a text expressing the contents with characters exists.
With the xe2x80x9cMethod And Device For Retrieving Video Imagexe2x80x9d disclosed in Japanese Patent Laid-Open Publication No. HEI 9-50436, a keyword for retrieval is assigned to each video information, so that by inputting a retrieval keyword indicating desired audio information, to retrieve video information with a keyword for retrieval identical to the retrieval keyword can be retrieved. However, it is necessary to assign a keyword for retrieval taking into considerations contents of the entire video information as an object for retrieval when assigning a keyword for retrieval to the video information, so that it is disadvantageously troublesome to select and assign a keyword for retrieval when there are may topics in one video information, or when a time of one video information is long.
The xe2x80x9cVideo Retrieval Methodxe2x80x9d disclosed in Japanese Patent Laid-Open Publication No. HEI 10-40260 requires that character information such as superimposition is included in the video information as an object for retrieval, and can not be applied to video information not including character information.
Similarly, with the conventional technology as described above, there has not been provided a method for expressing a logic structure inside an audio information. Therefore, there are the problems that it is not easy to use audio information as compared to other types of information, or that there are some restrictions in reproducing audio information. In other words, a user interface enabling sufficient improvement in operating, processing, and treating audio information is not known.
Further a video retrieval system making it possible to easily retrieve internal information inside video information has not been provided in the conventional technology.
It is the first object of the invention to provide a recording medium with video index information having a data structure capable of expressing a logic structure inside video information recorded therein.
It is the second object of the invention to provide a recording medium with video index information recorded therein with video retrieval improved by using the video index information having a data structure capable of expressing a logic structure inside the video information.
It is the third object of the invention to provide a technique for user interface making it possible to express a logic structure inside video information, to easily use the video information, to increase the freedom in reproducing the video information, and to improve adaptability of the video information to operation, application, and treatment.
It is the fourth object of the invention to provide a technique for user interface making it possible to raise a freedom in reproducing video information and to easily and effectively use video information.
It is the fifth object of the invention to provide a recording medium with audio index information having a data structure capable of expressing a logic structure of audio information recorded therein.
It is the sixth object of the invention to provide a recording medium with audio index information recorded therein with convenience in audio retrieval improved by using the audio index information having a data structure capable of expressing a logic structure inside audio information.
It is the seventh object of the present invention to provide a technique for user interface making it possible to express a logic structure inside audio information, to easily use the audio information, to raise a freedom in reproducing audio information, and to improve adaptability of audio information to operation, application, and treatment.
It is the eighth object of the present invention to provide a video retrieval method enabling improvement of convenience in video retrieval by using the video index information having a data structure capable of expressing a logic structure inside the video information and applicable to any type of video information.
It is the ninth object of the present invention to provide an audio retrieval method enabling improvement of convenience in audio retrieval using audio index information having a data structure capable of expressing a logic structure inside audio information.
It is the tenth object of the present invention to provide a video retrieval system making it possible to easily retrieve even inside information in video information by using video index information having a data structure capable of expressing a logic structure inside the video information.
The video information management method which uses the video index information according to the present invention is a video information management method which uses the video index information for managing video information by analyzing video information including at least a plurality of frames, generating apart from the video information video index information for managing the video information, and referring to the generated video index information, in which the video index information has a tree structure comprising at least one frame information for managing a given number of successive frames in video information as one unit of information, at least one sound information for managing sounds corresponding to a given range in the video information as one unit of information, at least one segment information for classifying frame information and sound information each as one unit of information and managing the information as a group having desired meanings, and package information for a plurality of segment information as a group each as a structure element object, and further in the tree structure, the segment information manages package information for managing other segment information prepared by using frames and sounds in the same range as that of the frame information and sound information managed by the segment information as a group, and package information is allocated in addition to frame information and sound information under one segment information and video information is managed by using a tree structure of video index information and structure element objects therein.
Further, the video information management method which uses the video index information according to the present invention is a video information management method which uses the video index information for managing video information by analyzing video information including at least a plurality of frames, generating apart from the video information video index information for managing the video information, and referring to the generated video index information, in which the video index information has view information consisting of at least one segment information for managing a range of a given number of successive frames in video information and a link list for segment information indicating in what order the plurality of segment information are to be followed, a plurality of link list each for indicating a desired order for following a required number of segment information are generated, a plurality of view information each corresponding to each link list are set, and video information is managed by using the video index information having a plurality of view information.
Further, with the recording medium with audio index information recorded therein according to the present invention, it is possible to provide a recording medium with the audio index information having a data structure capable of expressing a logic structure inside audio information and enabling improvement of convenience in audio retrieval.
Further, the audio information management method which uses the audio index information according to the present invention is an audio information management method which uses the audio index information for managing audio information by analyzing audio information including at least a plurality of frames, generating apart from the audio information audio index information for managing the audio information, and referring to the generated audio index information, in which the audio index information has a tree structure comprising at least one frame information for managing a given number of successive frames in audio information as one unit of information, at least one sound information for managing sounds corresponding to a given range in the audio information as one unit of information, at least one segment information for classifying frame information and sound information each as one unit of information and managing the information as a group having desired meanings, and package information for a plurality of segment information as a group each as a structure element object, and further in the tree structure, the segment information manages package information for managing other segment information prepared by using frames and sounds in the same range as that of the frame information and sound information managed by the segment information as a group, and package information is allocated in addition to frame information and sound information under one segment information and video information is managed by using a tree structure of video index information and structure element objects therein.
Further, in the video retrieval method using video index information according to the present invention, of the structure element objects, at least one type of structure element objects have retrieval information for retrieving contents of frames and/or sounds directly and indirectly managed by the structure element objects, further the segment information manages package information for managing other segment information prepared by using frames and sounds in the same range as that of the frame information and sound information managed by the segment information, and in the tree structure package information is allocated in addition to frame information and sound information under one segment information, retrieval conditions for a desired video-image (scene) are inputted to identify video index information including retrieval information satisfying the inputted retrieval conditions by retrieving a plurality of video index information previously prepared from a plurality of different video information, and a list for video information corresponding to the identified video index information is outputted as a result of retrieval.
Further, in the video retrieval method which uses the video index information according to the present invention, a structure element object has retrieval information for retrieving information for retrieving contents of frames and/or sounds directly or indirectly managed by the structure element object, and segment information manages package information for managing other segment information prepared by using frames and sounds in the same range as that of the frame information and sound information managed by the segment information, and in the tree structure, package information is allocated in addition to frame information and sound information under one segment information, retrieval conditions for a desired scene are inputted, a structure element object including retrieval information satisfying the retrieval conditions is identified by retrieving video index information, and a list for identified structure element objects is outputted as a result of retrieval.
Further, in the audio retrieval method which uses the audio index information according to the present invention, of the structure element objects, at least one type of structure element objects have retrieval information for retrieving contents of frames and/or sounds directly and indirectly managed by the structure element objects, further the segment information manages package information for managing other segment information prepared by using frames and sounds in the same range as that of the frame information and sound information managed by the segment information, and in the tree structure package information is allocated in addition to frame information and sound information under one segment information, retrieval conditions for desired sounds are inputted to identify audio index information including retrieval information satisfying the inputted retrieval conditions by retrieving a plurality of audio index information previously prepared from a plurality of different audio information, and a list for audio information corresponding to the identified audio index information is outputted as a result of retrieval.
Further, in the audio retrieval method which uses the audio index information according to the present invention, a structure element object has retrieval information for retrieving information for retrieving contents of sounds directly or indirectly managed by the structure element object, and segment information manages package information for managing other segment information prepared by using sounds in the same range as that of sound information managed by the segment information, and in the tree structure, package information is allocated in addition to sound information under one segment information, retrieval conditions for a desired scene are inputted and structure element object including retrieval information satisfying the retrieval conditions is identified by retrieving audio index information, and a list for identified structure element objects is outputted as a result of retrieval.
Further, the video retrieval system according to the present invention is a video retrieval system comprising a plurality of video information databases, a plurality of video index information databases, a retrieval information delivery file, and a retrieval device, and when retrieval conditions for a desired scene are inputted, the retrieval device identifies retrieval information coincident or similar to the retrieval conditions, and outputs the required video information according to video index information including the retrieval information as a result of retrieval. Therefore, it is possible to provided a video retrieval system in which retrieval reflecting even internal information of video information can easily be executed by using video index information having a data structure capable of expressing a logic structure inside the video information.